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UNIVERSAL INTERFACE FOR VOICE ACTIVATED ACCESS 
5 TO MULTIPLE INFORMATION PROVIDERS 

BACKGROUND OF THE INVENTION 
The present invention relates to an interface for 
accessing data and, more particularly, to a universal 

10 interface for accessing data from one or more information 
systems using a wireless telephone. 

As the content available from information providers 
expands, so does the complexity of retrieving data. For 
example, prior to the widespread use of electronic mail (e- 

15 mail) and alternate telecommunication systems, individuals 

would typically leave messages on and retrieve messages from 
telephone answering machines. With the numerous message 
centers available today, messages may be scattered over 
various telephone Messaging systems, electronic mail 

20 accounts, and facsimile -machines . Moreover, on a daily, if 
not hourly, basis, individuals are storing information on, 
and retrieving information from, personal information 
management systems (PICS) , electronic office managers,, and 
world wide web. sites. 

25 The availability of these information systems renders 

information, formerly hard, to manage and retrieve, readily 
accessible. Aiding the accessibility of this data is the 
fact that much of the data, some of which was previously 
available only in hardcopy, is now stored electronically and 

30 is thus searchable and retrievable. Therefore, individuals 
are able to access a multitude of data from multiple 
information providers and from the convenience of their 
computers . 

To date, the advantages associated with the widespread 
35 availability of electronic data have not been fully 

realized. At least three drawbacks exist with respect to 
the manner in which information currently is stored. First, 
not all information is stored or available in the same 
format. For example, some information is stored as speech, 
40 others are stored as text. Second, some information is 

available only in text form, and would thus be inaccessible 
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to a traveler or a mobile user who needs to access 
information using his or her wireless phone. Third, even 
the if information is available in the same format and 
accessible to wireless telephones, the information is not 
5 centralized, but is generally offered by a number of 
information providers. Thus, if one sought access to 
various data, he or she would need to access numerous 
information systems. 

10 SUMMARY OF THE INVENTION 

A universal interface accesses one or more information 
systems from a telephone, each of the information systems 
being adapted to generate text data or speech data in 
response to a query. The universal interface includes an 

15 interface control module for receiving a sound-based command 
from a user and for sending a corresponding machine readable 
command to a source selected from one of the information 
systems, the interface control module receiving data from ■ 
the source in response to the corresponding machine readable 

20 command; a text -to- speech converter coupled to the interface 
module for converting text data received by the interface 
control module to a converted speech segment and playing the 
converted speech segment for the user; and a router coupled 
to the interface module for receiving speech data from the 

25 source and playing a speech segment corresponding to the 
speech data for the user. 

Implementations of the system may include one or more 
of the following. The telephone can be wireless. The 
sound-based command can come from the user's voice, or can 

30 be specified using one or more telephone touch-tones. 

In another aspect, a universal interface accesses one 
or more information systems from a user telephone. The 
universal interface includes an input converter for 
converting input from the user telephone to commands; an 

35 interface control module coupled to the input converter for 
receiving the commands from the input converter; determining 
one of the information systems to be accessed; converting 
the commands to commands recognizable by the information 
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system; forwarding the converted commands to the information 
system; receiving data from the information system; 
detecting the form of the data from the information system; 
and storing information relating to a current state of the 
5 system. The universal interface also includes a 

speech- to- text routing switch coupled to the interface 
control module for receiving data from the information 
system and control data from the interface control module; a 
speech- to- command converter coupled to the interface control 

10 module for converting speech to commands, wherein the 

speech- to- command converter coupled to the speech- to -command 
routing switch to receive speech and to forward commands to 
the interface control module; an output switch coupled to 
the interface control module and the speech-to-text routing 

15 switch for receiving speech from the speech-to-text routing 
switch for receiving a control input from the interface 
control module, and for forwarding speech from the 
speech-to-text routing switch to the user telephone; and a 
text -to- speech converter coupled to the output switch for 

20 receiving text from the interface control module, converting 
the text to speech, and forwarding the speech to the output 
switch to deliver speech to the user telephone. The user 
telephone is a mobile telephone. The input converter can be 
a speech -to -command converter, a tone -to -command converter. 

25 The interface control module integrates the data from the 
information system periodically or manually under user 
control and includes: means for retrieving data from the 
information systems; means for determining antecedent 
comparable relevance of the data; means for updating all of 

30 the data to reflect the most recent data; means for linking 
relevant data; and means for exporting the linked data to 
the information system. 

Implementation of the above aspect may include one or 
more of the following. The information systems can include 

35 two or more of the following: a calendar; a to-do list; an 
address book; voice mail; e-mail; and a web site. The 
interface control module integrates and synchronizes (i)a 
database of a personal information manager, (ii) a database 
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residing on a personal digital assistant, and (iii) a 
database residing in the universal interface, the universal 
interface coupled to a computer on which the personal 
information database resides. A computer further comprises: 
5 a first input terminal for receiving data from the personal 
digital assistant; a second input terminal for receiving 
data from the universal interface; a sensor for detecting a 
synchronization event triggered by a user requesting 
synchronization of the database of the personal digital 

10 assistant with the database of the personal information 

manager; an electronic mail system coupled to the Internet; 
and control logic coupled to receive the synchronized event 
from the sensor and to transmit data, over the electronic 
mail system, to the interface control module, wherein the 

15 control logic updates the data in each of the databases to 
reflect the most recent data entered into any database. 
The synchronization information can be sent to the universal 
interface in its entirety, in compressed form, or in 
incremental form. The. data sent from the information system 

20 to the universal interface can be sent over the Internet. 
The data sent can be encrypted. The output switch can be a 
detector for determining whether the speech from the 
speech-to-text routing switch is digital or analog; a sound 
card coupled to the detector to receive digital data; a 

25 first speech routing switch for receiving a control input 
from the detector, and speech from the detector or speech 
from the sound card; a second speech routing switch for 
receiving the control input from the interface control 
module and speech from the first speech routing switch or 

30 speech from the text-to-speech converter, the second speech 
routing switch forwarding speech to the user telephone. The 
information system can include at least one of: a dual tone 
multiple frequency (DTMF) driven voice mail system, a voice 
driven voice mail system, an electronic mail system, a web 

35 site, and a personal information manager. The system can 
determine one of the following: whether the voice commands 
are being received from a user telephone; the information 
system to be accessed; whether the voice commands, after 
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being converted to commands, are recognizable to the 
information system; whether the converted commands have been 
forwarded to the information system; whether data has been 
received from the information system; whether data from the 
5 information system is speech or text; the state of the 

speech-to-text routing switch; and the state of the output 
switch. 

The interface control module further includes a model 
for converting commands from the telephone into commands 

10 recognizable by the information system; and a translator 
coupled to the input converter for retrieving the model 
corresponding to the information system to be accessed and 
for converting the commands to commands recognizable by the 
information system. The universal interface can include 

15 means coupled to the input converter for signaling that the 
user telephone has received unintelligible words; means for 
restarting communication to the text-to-speech converter, at 
a point a specified number of words back from the point at 
which the communication ceased; means for forwarding the 

20 first specified number of words by spelling the v/ords out; 
and means for continuing forwarding the data after the 
specific number of words. The universal interface can 
include means for detecting a first language in which the 
commands from the user telephone are received; means for 

25 detecting a second language associated with the data 
received from the information system; and means for 
converting the data from the information system into the 
first language. The system can detect more than one 
languages within a single fragment of data. A resource 

30 manager for establishing conference bridges to an external 
telephone having a telephone number, wherein the interface 
control module detects, from the commands from the input 
converter, whether a conference bridge request has been 
made; the interface control module retrieves the telephone 

35 number of the external telephone to establish the conference 
bridge; the interface control module forwards the telephone 
number to the resource manager; and the resource manager 
establishes a telephone connection with the external 
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telephone. A facsimile manager can be used for sending 
facsimiles to one or more facsimile machines, wherein: the 
interface control module detects, from the commands from the 
input converter, whether a facsimile request has been made; 
5 the interface control module retrieves a telephone number of 
a designated facsimile machine; the interface control module 
forwards the data and the telephone number to the facsimile 
manager; and the facsimile manager faxes the data to the 
designated facsimile machine . 

10 . The universal interface can have a pager manager for 

sending pager messages to one or more pagers, wherein the 
interface control module detects, from the commands from the 
input converter, whether a pager request has been made; the 
.interface control module retrieves data to be forwarded to 

15 the pager; the interface control module retrieves a 

telephone number of a designated pager to which the data is 
to be forwarded; the interface control module forwards the 
data and the telephone number to the pager manager; and the 
pager manager forwards the data to the designated pager. 

20 In another aspect, a universal interface for accessing 

data from one or more information systems includes an input 
converter for receiving inputs from the user telephone and 
converting the inputs into commands; and an interface 
control module coupled to the input converter. The 

25 interface control module includes means for periodically 
requesting data from the information systems; means for 
retrieving data from the information system; means for 
detecting the form of data from the information system; 
means for converting the data that is speech to commands; 

30 means for storing the data for later access by a user 

telephone; and means for storing information representing a 
current state of the system; and further including means for 
receiving commands from the input converter and means for 
detecting which of said stored information is sought to be 

35 . accessed; 

a text-to-speech converter coupled to the interface 
control module for receiving said requested stored text from 
the interface control module, converting the text to speech, 
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and forwarding the speech to the user telephone. 

The input converter can be a speech- to-command converter or 
a tone -to -command converter. The interface control module 
5 integrates the data from the information system periodically 
or manually under user control, and can include means for 
retrieving data from the information systems; means for 
determining antecedent comparable relevance of the data; 
means for updating all of the data to reflect the most 

10 recent data; means for linking relevant data; and means for 
exporting the linked data to the information system. 

The universal interface of claim 25, wherein the 
interface control module integrates and synchronizes (i) a 
database of a personal information manager, (ii) a database 

15 of a personal digital assistant, and (iii) a database 

residing in the universal interface, the universal interface 
coupled to a computer on which the personal information 
database resides. The computer can include a first input 
terminal for receiving data from the personal digital 

20 assistant; a second input terminal for receiving data from 
the universal interface; a sensor for detecting a 
synchronization event, wherein the event is triggered by a 
user requesting synchronization of the database of the 
personal digital assistant with the database of a personal 

25 information manager; an electronic mail system coupled to 
the Internet; control logic coupled to receive detection 
information from the sensor and to transmit data, via the 
electronic mail system, to the interface control module, 
wherein the control logic updates the data in each of the 

30 databases to reflect the most recent data entered into any 
one database. The synchronization information is sent to 
the universal interface in its entirety, in compressed form, 
or in incremental form. The data sent from the information 
system to the universal interface is sent over the Internet, 

35 The data sent over the Internet is encrypted. The 

information system can be a dual tone multiple frequency 
(DTMF) driven voice mail system, a voice driven voice mail 
system, an electronic mail system, a web site, and a 
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personal information manager. The system can determine 
whether the voice commands are being received from a user 
telephone; the information system to be accessed; whether 
the voice commands, after being converted to commands, are 
5 recognizable to the information system; whether the 

converted commands have been forwarded to the information 
system; whether data has been received from the information 
system; whether data from the information system is speech 
or text; the state of the speech-to-text routing switch; and 

10 the state of the output switch. The interface control 
module includes one or more models containing commands 
recognizable by the information system; control logic for 
accessing the model that corresponds with the information 
system to be accessed, converting control commands to 

15 commands recognizable by the information system, and 

forwarding the converted commands to the information system. 
The universal interface includes means coupled to the input 
converter for signaling that the user telephone has received 
unintelligible words; means for restarting communication to 

20 the text-to-speech converter at a point a specified number 
of words back from the point at which the communication 
ceased; means for forwarding the first specified number of 
words by spelling the words out; and means for continuing 
forwarding the data after the specific number of words. The 

25 universal interface can include means for detecting a first 
language in which the commands from the user telephone are 
received; means for detecting a second language associated 
with the data received from the information system; and 
means for converting the data from the information system 

30 into the first language. The universal interface further 

includes means for detecting more than one language within a 
single fragment of data. 

In another aspect, a method provides data from one or 
more information systems to a user telephone by converting 

35 speech from the user telephone to commands; determining the 
information system to be accessed; converting the text to 
commands recognizable by the information system; forwarding 
the converted commands to the information system; receiving 

8 
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data from the information system; detecting the form of the 
data from the information system; converting non- speech data 
from the information system into speech; forwarding the 
speech data to the user telephone; and storing information 
5 relating to the current state of the system. 

Implementations may include one or more of the 
following. The converting speech from the user telephone can 
include converting speech from a mobile telephone, 
converting speech to commands, or converting 

10 dual-tone-multiple-frequency (DTMF) information to commands. 
The data can be synchronized from the information systems by 
periodically, or upon command: retrieving data from the 
information systems; comparing the data to determine which 
data is most recent; updating all of the data to reflect the 

15 . most recent data; linking relevant data; and exporting the 
linked data to the information systems. 

The information systems can include two or more of the 
following: a calendar; a to-do list; an address book; voice 
mail; e-mail; and a web site. The method includes 

20 integrating and synchronizing (i) a database of a personal 
information manager, <ii) a database of a personal digital 
assistant, and (iii) a database residing in a universal 
interface, by: receiving data from the personal digital 
assistant; receiving data from the universal interface; 

25 detecting a synchronization event, wherein the event is 

triggered by a request for synchronization of the database 
of the personal digital assistant with the database of a 
personal information manager; transmitting data, via an 
electronic mail system, to the interface control module; and 

30 updating the data in each of the databases to reflect the 
most recent data entered into any one database. The 
synchronization information can be sent to the universal 
interface either in its entirety, in compressed form, or in 
incremental form. The data can be sent from the information 

35 system to the universal interface over the Internet. The 
method includes encrypting the data from the information 
system before sending the encrypted data over the Internet . 
The information system can include at least one of: a voice 
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mail system, and an electronic mail system, a web site, and 
a personal information manager. The method includes storing 
information relating to the current state of the system 
further including determining whether the voice commands are 
5 being received from a user telephone; the information system 
to be accessed; whether the voice commands, after being 
converted to commands, have been converted into commands 
recognizable by the information system; whether the 
converted commands have been forwarded to the information 

10 system; whether data has been received from the information 
system; whether data from the information system is speech 
or text; the state of the speech-to-text routing switch; and 
the state of the output switch. 

The converting of text to commands recognizable by the 

15 information system can include accessing one or more models 
for converting commands from the telephone into commands 
recognizable by the information system. The method also 
includes receiving a command from the user telephone 
signaling that the user telephone has received 

20 unintelligible words; ceasing communication to the user 

telephone; restarting communication to the user telephone at 
a point a specified number of words back from the point at 
which the communication ceased; forwarding the first 
specified number of words by spelling the words out; and 

25 continuing to forward the remainder of the data. The method 
can also include detecting the language in which the 
commands from the user telephone are received; detecting the 
language of the data received from the information system; 
converting the data from the information system into the 

30 language in which commands from the user telephone are 
received . 

The detecting the language of the data received from 
the information system further includes detecting more than 
one language within a single piece of data. The method 
35 includes establishing conference bridges to one or more 

external telephones, by: detecting, from the commands from 
the input converter, whether a conference bridge request has 
been made; retrieving a telephone number of the external 
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telephone to which the conference bridge is to be 
established; forwarding the telephone number to a resource 
manager; and establishing, through the resource manager, a 
telephone connection with the external telephone, thereby 
5 opening a line of communication between the user telephone 
and the external telephone. The method includes sending 
facsimiles to one or more facsimile machines, by: detecting, 
from the commands from the input converter, whether a 
facsimile request has been made; retrieving data to be 

10 faxed; retrieving a telephone number of a designated 

facsimile machine to which the data is to be forwarded; 
forwarding the data and the telephone number to a facsimile 
manager; and with the aid of the facsimile manager, faxing 
the data to the designated facsimile machine. The method 

15 also includes sending pager messages to one or more pagers, 
by: detecting, from the commands from the input converter, 
whether a pager request has been made; retrieving data to be 
forwarded to a pager; retrieving a telephone number of a 
designated pager to which the data is to be forwarded; 

20 forwarding the data and the telephone number to a pager 

manager; and with the aid of the pager manager, forwarding 
the data to the designated pager. 

In another aspect, a method accesses data from one or 
more information systems by retrieving data from the 

25 information system; detecting the form of data from the 

information system; converting the data that is speech to 
text; storing the data for later retrieval ; and storing 
information representing the current state of the system. 

Advantages of the invention include one or more of the 

30 following. The invention provides a seamless access to one 
or more information systems. More particularly, the 
invention provides access to multiple communication systems, 
some of which are text-driven, using a wireless telephone. 
The Universal Interface ("UI") of the invention recognizes 

35 voice commands, converts the commands into commands 

recognizable by the information systems, and utilizes the 
converted commands to access information systems and 
retrieve data therefrom. The UI enables an individual to 
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retrieve all of his or her information using a single device 
such as a wireless telephone. For instance, the individual 
can retrieve various information from a wireless telephone 
including voicemail messages, personal information manager 
5 information, to do lists, electronic mail messages, and 
information stored on the Internet . 

The UI presents the user with an abstraction of a 
unified mailbox (fax, e-mail, voice mail, among others) and 
advanced web services from a wireless telephone. It does 

10 not lock the individual into one information provider, and 
allows the user to access multiple information providers 
with a single telephone call. It also provides a simple 
system that acts as a conduit, translating an individual's 
spoken or tone commands into commands recognizable by the 

15 various information systems. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a block diagram illustrating a universal 
. interface for a mobile user. 
20 Fig. 2 is a block diagram showing the universal 

interface for accessing the Internet. 

Fig. 3 is a block diagram illustrating a first 
embodiment of the universal interface for processing speech 
commands . 

25 Fig. 4 is a block diagram illustrating a second 

embodiment of the universal interface for processing touch- 
tone input . 

Fig. 5 is a block diagram illustrating a third 
embodiment of the universal interface, with resource, 
30 facsimile, and pager managers. 

Fig. 6 is a block diagram showing the universal 
interface of Fig. 3 for communicating with a personal 
information manager ("PIM") . 

Fig. 7 is a block diagram illustrating an interface 
35 control module. 

Fig. 8 is a block diagram illustrating an output switch 
of the universal interface. 

Fig. 9 is a block diagram illustrating a speech-to-text 



12 



WO 01/76212 



PCT/USOO/01547 



routing switch of the universal interface. 

Fig. 10 is a block diagram illustrating various ways of 
accessing the information systems using the universal 
interface. 

5 Fig. 11 is a block diagram illustrating automatic data 

retrieval and non-real-time access to the universal 
interface using voice commands. 

Fig. 12 is a block diagram illustrating automatic data 
retrieval and non-real-time access to the universal 
10 interface using touch-tone inputs. Fig. 12 includes 
language converter and spell converter options. 



DESCRIPTION 

15 Referring now to Fig. 1, a universal interface 200 for 

mobile users is shown. A user communicates with an 
information system 110 using a telephone 105 which may be 
wireless. A universal interface 200 receives user commands 
from the wireless telephone 105 and converts those commands 

20 into commands recognizable by the information system 110. 

The universal interface 200 then sends those commands to the 
information system 110. The information system 110 responds 
to the user commands and sends the responsive information to 
the universal interface 200. The universal interface 200 

25 converts the data from the information system 110 if the 
data is not in a format recognizable by the wireless 
telephone 105. Finally, the universal interface 200 
provides the data to the mobile telephone 105, thus 
providing the data to the user. This process of retrieving 

3 0 information from the information system 110 through the 

universal interface 200 is repeated until the user retrieves 
all the required data. 

The universal interface 200 can be used to access 
various data in addition to voice mails and e-mail. For 

35 example, as shown in Fig. 2, the user can access Internet 

data such as stock quotes and weather reports from web sites 
125 and 130 on the Internet 120. These data are collected 
and processed by a web server using intelligent web server 
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agent tools such as the webMethods B2B , extensible Markup 
Language (XML) - based e-commerce solution, available from 
webMethods, Inc. of Fairfax, Virginia. 

WebMethods B2B automates data exchange between 
5 different applications, Web sites and legacy data sources. 
WebMethods B2B employs Web Interface Definition Language 
(WIDL) , which enables the characterization of automated 
accesses to Web-enabled resources and other applications 
through well-defined interfaces. By abstracting such 

10 information, the applications can continue to be accessed 
regardless of changes in the location, appearance, and 
structure of the underlying data. 

Alternatively, data may be collected and processed by a 
web server using VoxML Voice Mark-up Language, available 

15 from Motorola, Inc. VoxML is a mark-up language designed 

specifically for voice applications on the Web. It is based 
on the W3C XML standard and follows the syntactic rules of 
XML. Further, VoxML offers the advantage of allowing for 
the querying of web servers through the use of voice 

20 commands sent over a telephone. The VoxML language can be 
used to express multi-step dialogues that collect commands, 
input values, and audio samples from a user, and play 
recorded audio and synthesized speech back to the user. 

In one embodiment, to communicate with the user, the UI 

25 200 is connected to a cellular switch over one or more Tl 
lines. This connection allows a mobile telephone user to 
dial into the cellular switch, and concomitantly into the UI 
200. 

The UI 200 uses the D24 0 Tl platform, available from 
30 Dialogic Corp. of Parsippany, NJ. Connectivity to the 
cellular switch can be via SS7 or E&M MF/DTMF Channel 
Associated Switching, where the UI 200 can talk to the 
switch using SS7 point codes. Ports can be universal, so 
that they can function either as incoming or outgoing ports 
35 using dynamic reallocations. The universal interface 200 

may operate on a scalable and fault-tolerant computer system 
such as that disclosed in copending U.S. Application Serial 
No. 09/169,838, entitled "Server Pool for Clustered System," 
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filed October 9, 1998 and commonly assigned, hereby 
incorporated by reference; and as that disclosed in 
copending U.S. Application Serial No. 09/169,361, entitled 
"Fault Tolerant Bus for Clustered System," filed October 9, 
5 1998, and commonly assigned, hereby incorporated by 
reference. 

Fig. 3 illustrates the internal structure of the 
universal interface. If the input to the universal 
interface 200 is voice, the voice is first converted to 

10 computer-recognizable commands via a speech-to-command 
converter 220, The converter 220 uses the Speech 
Recognition Technology software, available from Nuance 
Communications of Menlo Park, California. The 
implementation can either be host-based or line card-based. 

15 Once the speech has been converted to computer- 

recognizable commands, the converted commands are then 
provided to an interface control module ("ICM") 300. The 
interface control module 300 receives commands from the 
speech- to- command converter 220 and determines which 

20 information system the user is attempting to access. It 

then converts the commands into commands recognizable by the 
information system 110. The interface control module 300 
then establishes contact with the information system 110. 
Once contact has been established, the interface control 

25 module sends converted commands to the information system. 
These converted commands are processed by the information 
system in the same manner as if the user was accessing the 
information system directly. The information system 110 
receives the commands and provides data in response. The 

30 data generated by the information system 110 is provided to 
both the interface control module 300 and a speech-to-text 
routing switch 500. The interface control module 300 
determines whether the data from the information system 110 
is speech or text. The interface control module 300 then 

35 sends a control signal to the speech-to-text routing switch 
5.00. This control signal indicates whether the speech-to- 
text routing switch 500 should be toggled. 

In a toggled state, the data from the information 
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system 110 is sent directly through the speech-to-text 
routing switch 500 into an output switch 400. The speech- 
to-text routing switch 500 is only toggled, and thus only 
sends data directly to output switch 4 00, when the interface 
5 control module 3 00 detects that the data from the 

information system 110 is speech. The data sent directly to 
output switch 400 is also sent to speech- to- command 
converter 260. The speech- to- command converter 260 converts 
the speech into commands and sends the commands to the 

10 interface control module 300. The ICM 300 uses these 
commands to determine the state of the system. 

If, on the other hand, the interface control module 3 00 
detects that the data from the information system 110 is 
text, the interface control module will route the text to 

15 the text -to- speech converter 230, and will send a control 
signal to output switch 400, routing the speech, from the 
text-to-speech converter 230 to the output of the universal 
interface 200. 

The text-to-speech converter 230 automatically produces 
20 speech through transcription. The text-to-speech converter 
contains two modules- -one that processes natural language 
and a second that performs digital signal processing. The 
first module converts text into its phonetic equivalent. 
The second converts the output of the first into speech. 
25 In the alternative, a conventional voice response system can 
be employed. 

Also, in the embodiment the text -to- speech engine may 
operate on a D240 Tl line card (which is available from 
Dialogic). Also, Accuvoice's or Lernout & Hauspie's 

30 TruSpeech software can be used to handle speech. 

Also, in one embodiment, a VUI (Voice User Interface) 
indicates various phrases that the system will recognize, 
and a VAV (Virtual Assistant Voice) indicates prompts and 
information from voice messages, PIM or the Web that the 

35 system will play. The VUI allows the user to speak 

naturally to request the information the user needs at any 
level. However, when the user wants to enter information to 
the speaker, the user initiates a step-by-step process to 
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interact with the system. 

For example, a user's first command may take the form 

of: 

W I would like to access my [menu item]," where menu item 
5 might be calendar, to do list, address book, auto wakeup, 
web information, or another information system, where words 
not between brackets are optional. 

Once the user has accessed one of the menu items, 
typical user commands might include: 
10 Plav mail : u [Play, Brief ] [priority] [E-mail, voice 

mail] from [person]," where priority can be urgent or 
normal, among others. 

Address book look-up : W I would like to [Find, lookup] 
[name]," where name is the person's name the user wants to 
15 find. 

Call management : "[I would like to] [call, dial, make a 
conference call to] [Name, phone number] , " where name is the 
person's name the user wants to call. 

Appointment and to do list lookup : W I would like to 
20 [find, check] My [appointment, to do] for [date]," where 
date can be tomorrow or any day of the week. 

When the user utters these commands to the system, the 
system looks for information to complete the commands. If 
the system cannot complete the commands, the UI 200 prompts 
25 the user step-by-step for the required information. For 

example if the system registers the appointment keyword, but 
not the date, the UI 200 prompts the user for only the date. 
The UI 200 might state, "Which appointment date do you want 
me to check?" 

30 Just as the UI 200 allows the user to access 

information, it operates similarly to allow the user to 
communicate information. In one embodiment, the UI 200 has a 
second level menu structure. This second level structure, 
in response to the user's commands, prompts the user for 

35 information. For example, the initial user command might 
be: H I would like to [make, delete, update] [an entry, an 
item, a name] . " If the user chooses to update an address 
entry, the UI 200 may prompt the user as follows: 
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"What is the business phone number of the person?" 
[user speaks business phone number] 
"What is the home phone number of the person?" 
[user speaks home phone number] 
5 In one embodiment, the UI 200 can repeat the user 

information to ensure that the information has been entered 
correctly. In another, the user can query the number of 
entries in the address book. 

The user can also enter entries to an appointment 
10 calendar. In this instance, the UI 200 may prompt the user 
for a date of the appointment, a beginning time of the 
appointment, a duration of the appointment, and an agenda. 

The user can also provide entries to the to-do list. 
The UI 200 might then provide prompts for the due date, 
15 priority, and agenda. If the user prompts the UI 200 to 

enter an autowakeup, the UI 200 may prompt the user for the 
date of the entry and the time of the entry, and the phone 
number to dial at the wake -up time. 

If the user prompts the UI 200 to set up a conference 
20 call, the UI 200 may prompt the user as follows: "Who do you 
want to conference in? Say the name of the persons or phone 
number. When you are finished say *done,'" After each time 
the user speaks a name or number, the UI 200 responds with 
"Got it. Who is next?" The UI 200 continues to prompt the 
25 user until the user indicates that he or she is done 
entering names or numbers. 

In another embodiment, if an incoming call occurs 
during the conference call, the UI 200 indicates to the user 
that there is an incoming call. For example, the UI 200 may 
30 say "[Name] is on the other line, do you want to take it?" 
The user may then answer: "[I will take it, conference her 
in, voice mail, (ignore virtual assistant)]." 

Just as the user can update information, he or she can 
also delete or modify entries. If the user attempts to 
35 delete information, the UI 200 may prompt the user to verify 
the request prior to deleting the entry. And, just as in 
conventional systems, certain use preferences such as 
whether or not to inquire before deleting may be stored in 
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memory . 

Further, once the user has prompted the UI 200 to 
perform a particular function, the UI 200 can prompt the 
user to ensure that the user does not want to repeat a 
5 similar function. For example, once the user has prompted 
the UI 200 to enter an item into the to-do list, the UI 200 
might prompt the user to determine if the user wants to add 
another entry. 

In one embodiment, the UI 200 has additional functions 

10 related to the playing or briefing of messages. These 

commands include: save, delete, play, forward, reply, next, 
previous, higher volume, lower volume, skip forward, skip 
backward, faster, slower. 

In Fig. 3, the text-to-speech converter 230 routes 

15 speech to the output switch 400. The interface control 

module 300 sends a control signal to the output switch 400. 
In a non-toggled state, the output switch 400 receives the 
output of the text-to-speech converter 230. In a toggled 
state, the output of the output switch 400 receives the 

20 output of the speech-to-text routing switch 500. 

If the data from the information system 110 is speech, 
it is routed directly to the output of the universal 
interface 200. If, on the other hand, the data from the 
information system 110 is text, the data is converted to 

25 speech via the text-to-speech converter 230 and is then 

routed through the output switch 400 to the output of the 
universal interface 200. 

Fig. 4 is similar to Fig. 3, except that the input to 
the universal interface 200 handles touch- tone inputs rather 

30 than voice inputs. Therefore, the speech- to- command 

converter 220 of Fig. 4 is replaced with a tone -to- command 
converter 250 of Fig. 4. The tone -to- command converter 250 
converts touch-tone inputs from a telephone of the user into 
commands. The commands are presented to the interface 

35 control module 300. The interface control module 300 then 
functions as described above . 

Fig. 5 illustrates a universal interface with resource, 
facsimile, and pager managers. These managers are 
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accessible from the web. 

The universal interface 200 accepts touch-tone from the 
user's telephone. This tone is converted subsequently to 
commands through the tone-to-command converter 250. As in 
5 Fig. 4, the output of the tone -to- command converter 250 is 
then sent to the interface control module 3 00. The 
universal interface 200 functions in the same manner as the 
universal interface 200 of Fig. 4, except that the universal 
interface 200 of Fig. 5 can support resources, facsimiles 

10 and pagers through a resource manager 270, a facsimile 
manager 280, and a pager manager 290. These managers 
interface with an external telephone 275, a facsimile 
machine 285, and a pager 295, respectively. 

The pager manager 290 can receive pages from another 

15 user, and can also page one or more external pagers 295. 
The pager manager 2 90 sends the information received from 
the pager 295 to the interface control module 300. . The 
pager manager manages pager and message delivery queues. 
The CJI 200 may use a native NT file system or CIFS virtual 

20 file system to manage system wide pager and dial-out queues. 
See, for example, copending U.S. Application Serial No. 
09/169,360, entitled "Shared-Everything File Storage for 
Cluster System," filed October 9, 1998, and commonly 
assigned, hereby incorporated by reference. 

25 Similarly, the facsimile manager 280 receives facsimile 

messages from another user, and sends faxes to the facsimile 
285. The fax manager manages the queues for facsimiles. It 
keeps track of the number of facsimile resources available 
in the system, maintains a list of outgoing faxes to be 

30 delivered, and incorporates a retry mechanism in case of an 
error . 

The UI 200 has a fax-only mailbox so that callers can 
deposit directly a fax into their recipient's mailbox 
without first ringing the handset of the recipient. A 
35 caller can dial the fax-only mailbox number and press a 
start button on his or her fax machine. This feature is 
referred to as walk-away- fax. 

The resource manager 270 assigns various conference 
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resources in a multi-node system, where a multi-node system 
can consist of multiple UI's within a single UI, or can 
consist of a number of UI's connected in tandem. The 
resource manager 270 provides inter-node time slot 
5 management for conference calls. 

As with the pager manager 280, the facsimile manager 
280 and the resource manager 270 communicate data from the 
external facsimile machine 285 and telephone 275, 
respectively, to the interface control module 300. 

10 Information from all three managers, 270, 280 and 290, 

becomes available to the interface control module 300 for 
later access by the user. Although Fig. 6 illustrates an 
embodiment in which the user communicates via a touch- tone 
pad, resource managers, facsimile managers and pager 

15 managers can also be incorporated into a system in which a 
user communicates via speech. 

To support the management of facsimiles, the universal 
interface 200 must contain facsimile drivers. The UI 200 
provides fax capabilities by integrating facsimiles through 

20 the use of the CP12 board available from Dialogic. The 
board and drivers may be interfaced through a fax 
application manager. 

Fig. 6 illustrates the universal interface with 
personal information manager PIM synchronization 

25 capabilities. The PIM 265 manages a user's address book, 
calendar, and to-do list. 

The address book contains entries with fields such as 
name, title, company address, business phone, home address, 
home phone, mobile phone, facsimile number, e-mail address, 

30 web page address and notes. 

The calendar keeps track of appointments. It has 
features such as subject/agenda, location, date/time, and 
appointment length. 

The to-do list is used to keep track of tasks that need 

35 to be accomplished. Each to-do list entry has fields such 
as subject /agenda, priority, due date/time, and reminder 
notes. The UI can page, call, or send e-mail with to-do 
list contents. 
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In the embodiment of Fig. 6, the user communicates with 
the universal interface 200, and therefore with the 
information systems, via a touch- tone pad. This embodiment 
differs from the embodiment of Fig. 4, in that the 
5 information system is a computer 262, to which a PIM 265 can 
be connected. 

In most PIM's, such as the "PalmPilot" or "Palm III," 
available from 3Com Corporation of Santa Clara, California, 
a synchronization process takes place from time to time with 

10 the personal computer 600. Using the "PalmPilot" as an 
example, this synchronization process is accomplished by 
placing the "PalmPilot" in a cradle, which is connected to 
the computer 252 through a serial cable; and pressing a 
"HotSync" button on the cradle. The synchronization process 

15 sends information from the "PalmPilot" device to the 

computer 262, thereby synchronizing databases on both units. 

In one embodiment, the UI 200 synchronizes the PIM 265 
with the universal interface 200 at the same time the user 
is synchronizing the PIM 265 with computer 262. In this 

20* embodiment, the UI 200 senses or intercepts the 

synchronization signal. When the UI 200 detects that the 
PIM 265 is requesting synchronization, the UI 200 
communicates with the computer 262 and imports the 
synchronization information therefrom. In this way, a 

25 remote server connection is made any time there is a 

synchronization event, thereby making the most current 
synchronized database available via a wireless phone. 

Alternatively, the universal interface 200 can 
periodically synchronize its database with the computer 262. 

30 In this embodiment, the universal interface 200 periodically 
communicates with computer 262 and requests information on 
the last PIM 265 synchronization. If the computer *262 
indicates that it has synchronized with the PIM 265 since 
the last communication with the universal interface 200, the 

35 universal interface 200 sends control logic to the computer 
262 to retrieve the most recent data. 

The personal information synchronization can take place 
over the web. Further, the data sent via the Internet can 
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be encrypted. Additionally, the data may be sent in 
compressed form or in incremental form when only new or 
updated data is transmitted. 

Fig. 7 illustrates in block diagram form, the structure 
5 of the ICM 3 00. In the embodiment of Fig. 7, control code 
310 receives as input commands from the speech-to-command 
converter 220 of Fig. 3 or the tone -to -command converter 250 
of Fig. 4. The control code 310 detects the type of 
information system 110 to be accessed, and then accesses a 

10 model 1 through N 320, 330, which corresponds with the 
particular information system 110. Each of the model 1 
through N 320, 330 provides a mapping of the input commands 
to commands recognized by the information system 110. The 
control code 310 looks up the chosen model, 320 or 330, the 

15 information system command that responds to the user input. 
The control code 310 then sends the command to the 
information system 110 and retrieves information therefrom. 

Figs. 8 and 9 illustrate in more detail the output 
switch 400 and the speech-to-text routing switch 500, 

20 respectively. The output switch 400 shown in Fig. 8 has 
three inputs. The first input receives, speech from the 
speech-to-text routing switch 500 (Figs. 4-5). This input 
represents speech data retrieved from the information system 
110. The second input receives the speech output of the 

25 text -to- speech converter 230. This input is used by the 
output switch 400 if the data from the information system 
110 is text which needs to be converted to speech by the 
text-to-speech converter 230. The third input to the output 
switch 400 is a switch control. If the interface control 

30 module 300 detects that the data coming from the information 
system 110 is text, it sends a switch control signal to 
switch 440, toggling switch 440 and allowing the text from 
the information system 110, after being converted by the 
text-to-speech converter 230, to flow to the output of the 

35 universal interface 200. Otherwise, if the interface 
control module 300 detects that the data from the 
information system 110 is speech, then the interface control 
module 300 sends a switch control signal to the switch 440 
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which sets the switch 44 0 in an untoggled state, thus 
routing the speech from the information system 110 through 
to the output of the universal interface 200. 

When the speech first enters the output switch 400, it 
5 is received by a detector 410. The detector 410 detects 
whether the speech is analog or digital in form. If the 
speech is in an analog form, the detector 410 sends the 
speech to switch 430. It also asserts the switch control 
input to switch 430, throwing the switch to a position that 

10 allows the speech input into switch 430 to be sent to the 
switch 440. If, on the other hand, the speech received by 
the detector 410 is digital, the digital speech is sent to 
sound card 420. At the same time, the detector 410 sends a 
switch control input to the switch 430, routing the output 

15 from sound card 420 through to switch 440. 

Fig. 9 illustrates the speech-to-text routing switch 
500. This speech-to-text routing switch 500 receives data 
from the information system 110. This data can either be in 
text or speech format. If the data from the information 

20 system 110 is in speech format, the speech-to-text routing 
switch 500 routes the speech directly to the output switch 
400. If, on the other hand, the data from the information 
system is in text format, the speech- to- text routing switch 
500 is opened. In this instance, the text data is not 

25 routed to the output switch 400 through the speech-to-text 
routing switch 500. 

The interface control module 300 controls the state of 
the switch 520 in the speech- to- text routing switch 500. In 
the event that the interface control module 3 00 detects that 

3 0 the data from the information system 110 is speech, it sends 
a control signal to the speech-to-text routing switch 500, 
throwing the switch 520 to a closed state. When the 
interface control module 300 detects text data from the 
information system 110, it sends the control signal to the 

35 speech-to-text routing switch 500, to maintain the switch 
520 in an open position. When the data from information 
system 110 is text, the text from the information system 110 
is received by the interface control module 3 00 and routed 
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through the text -to- speech converter 230 to the output 
switch 400, This text, which has been converted to speech 
by the text-to-speech converter 230, is then sent to the 
output of the universal interface 200. 
5 Turning now to Fig. 10, different methods in which a 

user can access various information systems 110 and 650, 
through various universal interfaces 200 and 620 are 
illustrated. In one embodiment, the user can access the 
information system 110 directly through a universal 

10 interface 200 through the use of a wireless telephone 105. 
In another embodiment, the universal interface 620 resides 
in a telephony server 610. In this embodiment, the 
universal interface 620 can access an information system 
650. The universal interface 620 retrieves information from 

15 the information system 650 periodically. When the user 
communicates with the telephony server 610 through a 
wireless telephone 670, the user can retrieve previously 
retrieved information from the information system 650. This 
previously retrieved information is stored in memory 660. 

20 Alternatively, and in a third embodiment, the user can 
interface with. the telephony server 610 to retrieve 
information in real-time. In this embodiment, the telephony 
server 610 acts as a conduit to allow connection between the 
wireless telephone 670 and the universal interface 200. 

25 From the user's standpoint, the operation is the same as if 
the user had accessed universal interface 200 directly. 

As previously discussed, the universal interface 200 
can operate in a nonreal-time mode. This mode is 
illustrated in Figs. 11 and 12. The interface control 

3 0 module 300 communicates with one or more information systems 
110 and 710. The interface control module 300 periodically 
retrieves data from these information systems. The 
interface control module 300 then stores the retrieved data 
in an internal or an external memory 660. If data retrieved 

35 from the information system 110 or 710 is in speech format, 
the interface control module 300 converts the speech to text 
before storing the data in memory 660. Once the data is 
stored in memory 660, the user can access previously stored 
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data at any time. In the particular embodiment illustrated 
in Fig. 12, the user may access the universal interface 
using a touch- tone pad. In this case, the touch- tone input 
is converted to text via a tone -to- command converter 250 and 
is then input into the interface control module 300. The 
interface control module 300 interprets the commands from 
the user and retrieves the appropriate data from memory 660. 
Because the data in memory 660 is in text format, the 
interface control module 300 sends the data to a text-to- 
speech converter 230. This text-to-speech converter 230 
converts the data into speech and provides the data in 
speech format to the user. 

The embodiment of Fig. 11 operates similarly to that in 
Fig. 12 except that the UI 200 is designed to receive speech 
as input. 

In one embodiment of the invention, the universal 
interface contains a language converter 720 (Fig. 12), which 
has the additional capability of translating electronic mail 
messages, and other data, which is stored in a different 
language, such as Spanish, French, among others. The . 
language converter 720 first sends the first few lines of an 
e-mail or message to a module called the "language 
identifier." The language identifier analyzes the words 
used by looking them up in a list of often used and typical 
words for all languages under consideration. For example, 
words often used in the English language might include 
'"the," "and," and "in." The converter tallies the results, 
and determines the appropriate language. 

If the language used is different from that spoken by 
the user (whose language can be ascertained by the language 
identifier), then the language converter 720, before 
communicating the language to the user, converts the data to 
the user's language. 

The universal interface may also monitor the language 
in which the data is stored. If the language converter 720 
detects a change in the language, the universal interface 
200 back-tracks a specified number of words and attempts to 
provide information in the language spoken by the user. 
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This process of changing languages mid- stream can occur 
multiple times within one transmission. 

In another embodiment, the universal interface accesses 
a spell converter 720 which spells out words that the user 
indicates are unintelligible. This embodiment compensates 
for pronunciation errors by the text -to -speech converters. 
These errors may occur because either the correct 
pronunciation falls into some grammatical exception or 
because the text being converted contains typographical 
errors . 

In this embodiment, the universal interface receives a 
command such as a pound (#) touch-tone from the user 
telephone indicating that the user has received 
unintelligible words. The universal interface 200 halts the 
text -to- speech converter 23 0 and restarts communication at a 
retry point a specified number of words back from stopping 
- point where the communication had ceased. When the 
universal interface 200 resumes communication, it accesses 
the spell converter 730 which tracks the text that was sent 
to the text-to-speech converter. The spell converter 730 
provides, to the Universal Interface, the spelling for a 
predetermined number of words. The universal interface 200 
then starts out by spelling a predetermined number of words 
from the restart point. The universal interface 200 then 
resumes communication at or near the same point where the 
user had indicated that the speech was unintelligible. Both 
the language converter 72 0 and the spell converter 73 0 may 
be modules internal to the interface control module 300. 

As discussed above, depending upon the user's needs, 
the universal interface 2 00 can retrieve information from 
one or more information systems. These systems may include, 
for example, electronic mail, voice mail, websites, or 
personal information managers. 

Regardless of the information system type, the 
universal interface 200 operates with one or more servers to 
provide the appropriate access. An e-mail server sends and 
receives messages using Internet Message Access Protocol 
(IMAP4) and Post Office Protocol/Simple Mail Transfer 
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Protocol (POP3/SMTP) protocols, where SMTP is the Internet 
standard for transmitting and forwarding messages, POP is 
the Internet standard for retrieving e-mail, and IMAP is the 
"next generation" Internet standard for retrieving e-mail. 
5 The e-mail server also services Internet voice and fax 
messages. 

In one embodiment, the e-mail server uses the same 
message store that is used for voice messages. Further, 
shared messages are not replicated. 

10 IMAP stands for Internet Message Access Protocol. It 

is a method of accessing e-mail or bulletin board messages 
that are kept on a mail server. It permits a "client" e- 
mail program to access remote message stores as if they were 
local. E-mail stored on an IMAP4 server can be manipulated 

15 from a desktop computer at home, a workstation at the 

office, and a notebook computer while traveling, without the 
need to transfer messages or files back and forth between 
these computers. IMAP's ability to access messages (both 
new and saved) from more than one computer is important as 

20 reliance on electronic Messaging and use of multiple 

computers is a common mode of operation for the mobile user. 
IMAP4 supports an offline access to the message store 
followed by a resynchronization of the message store with 
the server. 

25 The IMAP4 protocol includes operations for creating, 

deleting, and renaming mailboxes; checking for new messages; 
permanently removing messages; setting and clearing flags; 
server -based RFC- 822 and MIME parsing, and searching; and 
selective fetching of message attributes, texts, and 

30 portions thereof for efficiency. Note that MIME is an 

Internet standard for Messaging formatting. MIME allows for 
the attachment of various files to e-mail messages. These 
attachments might include text, voicemail, or faxes. 

The e-mail directory server utilizes a X.500 directory 

35 that stores user addresses, provides routing information, 
and unifies e-mail components in the environment. The e- 
mail directory server uses the following protocols to 
provide a distributed directory service: LDAP (Lightweight 
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Directory Access Protocol) for client access- -allows for the 
publishing and access of directories such as phone 
directories, DAP (Directory Access Protocol) for message 
routing information, DSP (Directory System Protocol) for 
5 forwarding requests to remote directory servers and 

obtaining results, DISP (Directory Information Shadowing 
Protocol) for replicating information between servers, and 
HTTP (one of two standard foundations for the World Wide 
Web) for access from Web browsers. The Web/Directory 

10 gateway facilitates access to LDAP and X.500 Directory 

information from a standard web browser, and utilities to 
import and export data to other applications, including user 
registration from other mail systems. 

Mailbox management must be reconciled between the voice 

15 and e-mail mailbox utilities. The E-mail X.500 mailbox 

structure and name service must be preserved, as they are 
integral to e-mail standards and protocols. In one 
embodiment, the voice mailbox tools are used as the primary 
mailbox management utilities. Voice mailbox creation would 

20 be exported to the X.500 e-mail box schema. Although there 
will be replication of the mailbox data between the voice 
and e-mail boxed structures, the message store is common 
between the voice and e-mail servers. Further, event 
notifications may be provided for messages received while 

2*5 the user is in a session. 

Another server may service the world wide web. Using 
Active X and Java applets and script technology, one can 
access information over the Internet using a standard web 
browser. The information accessible over the web includes 

30 information associated with Internet web sites such as 
stock, weather, business information, as well as 
synchronized information from the user's PIM, such as 
calendar, address book and to-do list. 

The web server uses COM interfaces to retrieve PIM 

35 information, e-mail, voice mail and fax messages. These 
operations may be performed locally on the web server. 

In an embodiment which handles voice mail systems, the 
UI 200 provides an interface to various legacy voice mail 
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systems such as systems from Octel or Centigram. The legacy 
system may forward voice and fax messages to the UI 200 
using VPIM (Voice Profile for Internet Messaging) . VPIM is 
based on MIME, an Internet standard for message formatting, 
5 which allows voice and fax mail systems to exchange messages 
over the Internet . 

The UI may have fewer voice mail features than the 
legacy systems . In such a case, the UI maps strings and 
features that are supported and accesses data according to 

10 its perhaps limited abilities. 

The UI provides the use of a voice or DTMF dial -out to 
the legacy system while preserving the virtual assistant 
functionality. The user can change parameters of the 
universal interface, and can also expand the UI 

15 capabilities, through either the telephone or the Web. 

Configuration changes over the web may be either menu or 
form driven. Changes over the phone may be menu driven. 

It should be noted that although the input to the 
universal interface 200 has been illustrated as either tone 

2 0 or speech, the invention can accept as input either speech, 
tone, or both speech and tone. Further, while the invention 
has been shown and described with reference to an embodiment 
thereof, those skilled in the art will understand that the 
above and other changes in form and detail may be made 

25 without departing from the spirit and scope of the following 
claims . 
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